Information processing apparatus and information processing method

ABSTRACT

A capturing parameter and a capturing image are obtained by an image capturing device which uses the capturing parameter. Correction data, which corresponds to an optical transfer function of the image capturing device derived from the capturing parameter, and a noise amount of the capturing image dependent on the capturing parameter, is acquired to correct a blur of the capturing image. A first degree of correction by the correction data for a high noise amount is less than a second degree of correction by the correction data for a low noise amount.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and imageprocessing method which correct a blur of a capturing image caused by animaging optical system.

2. Description of the Related Art

In an image capturing apparatus such as a digital still camera ordigital video camera, an imaging optical system formed from a lens andthe like guides light from an object onto a CCD or CMOS sensor servingas an image capturing device so as to form an image. The image capturingdevice converts the received light into an electrical signal. Theelectrical signal undergoes processing such as analog-to-digital (A/D)conversion and demosaicing necessary for converting an electrical signalinto image data, thereby obtaining a capturing image.

Since the light arriving at the image capturing device has passedthrough the imaging optical system, the image capturing deviceinfluences the image quality of the capturing image. For example, when ahigh-power lens is used, a capturing image having a high resolution toits peripheral region can be obtained. On the other hand, if a low-costlow-power lens is used, the resolution of the capturing imageconspicuously degrades especially in the peripheral region.

For example, when taking a picture of a starry sky, an image captured bya high-power lens shows each star almost as a point image. In an imagecaptured by a low-power lens, however, each star does not appear as apoint image but blurs. When taking a picture of a person, the use of ahigh-power lens enables one to obtain an image in which each hair isdistinct. However, a low-power lens only obtains an image with a blurrymass of hair. That is, an image without definition is obtained by thelow-power lens.

Such a blur depends on the characteristics of the imaging optical systemand occurs even in an in-focus state. In other words, the resolution ofthe capturing image changes depending on the performance of the lenseven in an in-focus state.

There is a method of correcting an image blur caused by the imagingoptical system by performing image processing for the capturing image.In this method, image processing is executed based on the blurcharacteristic of the imaging optical system, which has been acquired asdata in advance, thereby correcting an image blur caused by the imagingoptical system.

To obtain the blur characteristic of the imaging optical system as data,for example, a PSF (Point Spread Function) is used. The PSF representshow a point of an object blurs. For example, a luminous body (pointsource) having a very small volume is captured via an imaging opticalsystem in darkness. If an ideal imaging optical system is used, thelight forms a point image on the surface (light-receiving surface) ofthe image capturing device. However, if an imaging optical system thatyields a large blur is used, the light forms not a point image but animage spreading to some extent on the light-receiving surface. That is,the two-dimensional distribution of the light on the light-receivingsurface corresponds to the PSF of the imaging optical system. Whenacquiring the PSF of the imaging optical system in actuality, it is notalways necessary to capture an object such as a point source. Based onan image obtained by capturing, for example, a chart with white andblack edges, the PSF can be calculated using a calculation methodcorresponding to the chart.

A method of using an inverse filter is known as a method of correctingan image blur by the PSF. Assume a case in which a point source iscaptured in darkness for the descriptive convenience. When an imagingoptical system with a blur is used, light emitted by the point sourceforms a light distribution spreading to some extent on thelight-receiving surface. The image capturing device converts the lightinto an electrical signal. The electrical signal is converted into imagedata, thereby obtaining a digital image of the point source. In theimage obtained using the imaging optical system with a blur, not onlyone pixel corresponding to the point source has a significant, non-zeropixel value. Pixels in the neighborhood also have significant pixelvalues close to zero. Image processing of converting this image into animage having significant pixel values almost at one point is inversefiltering. The inverse filter allows to obtain an image as if it werecaptured using an imaging optical system with little blur.

A point source has been exemplified above for the sake of explanation.Regarding light from an object as an aggregate of a number of pointsources, the blur of light coming from or reflected by each point of theobject is eliminated, thereby obtaining an image with little blur evenfor an object other than a point source.

A detailed method of forming the inverse filter will be described next.A capturing image obtained using an ideal blur-free imaging opticalsystem is defined as f(x,y), where x and y are two-dimensional pixelpositions on the image, and f(x,y) is the pixel value of the pixel(x,y). A capturing image obtained using an imaging optical system with ablur is defined as g(x,y). The PSF of the imaging optical system with ablur is represented by h(x,y). Then, f, g, and h have a relation givenby

g(x,y)=h(x,y)*f(x,y)  (1)

where * represents a convolution operation.

Correcting an image blur (to be referred to as blur correctionhereinafter) amounts to estimating f obtained by the blur-free imagingoptical system based on the image g obtained using the imaging opticalsystem with a blur and h that is the PSF of the imaging optical system.When Fourier transformation is applied to equation (1) to obtain anexpression in the spatial frequency domain, it can be written as aproduct for each frequency as given by

G(u,v)=H(u,v)·F(u,v)  (2)

where

H is an OTF (Optical Transfer Function) that is the Fourier transform ofthe PSF,

u is the spatial frequency in the x direction,

v is the spatial frequency in the y direction,

G is the Fourier transform of g, and

F is the Fourier transform of f.

To obtain the blur-free capturing image f from the blurred capturingimage g, both sides of equation (2) are divided by H as per

G(u,v)/H(u,v)=F(u,v)  (3)

F(u,v) obtained by equation (3) is inversely Fourier-transformed to thereal space to obtain the blur-free image f(x,y). Let R be the inverseFourier transform of 1/H. Convolution is performed in the real space toobtain a blur-free image by

g(x,y)*R(x,y)=f(x,y)  (4)

R(x,y) of equation (4) is called an inverse filter. Actually, since adivision by a divisor 0 occurs at the frequency (u,v) at which H(u,v) iszero, the inverse filter R(x,y) requires a slight modification.

Normally, the higher the frequency is, the smaller the value of the OTFis. The higher the frequency is, the larger the value of the inversefilter that is the reciprocal of the OTF is. Hence, convolution of thecapturing image using the inverse filter enhances the high-frequencycomponents of the capturing image, that is, noise (noise is ahigh-frequency component in general) contained in the capturing image. Amethod is known to impart a characteristic which does not enhancehigh-frequency components as much as the inverse filter by modifyingR(x,y). A Wiener filter is famous as a filter that does not enhancehigh-frequency components so much in consideration of noise reduction.

As described above, it is impossible to completely remove a blur becauseof noise contained in the capturing image or a departure from an idealcondition such as existence of a frequency at which the OTF is zero.However, the above-described processing can reduce the blur. Note thatfilters such as an inverse filter and a Wiener filter to be used forblur correction will be referred to together as a “recovery filter”hereinafter. As a characteristic feature of the recovery filter, itperforms image processing using the PSF of the imaging optical system,as described above.

To construct a Wiener filter, the OTF and noise amount of the imagingoptical system are used. That is, as a characteristic feature, theWiener filter is formed using the PSF or OTF and noise amount of theimaging optical system.

Note that the inverse filter and Wiener filter have been mentioned asthe method of correcting a blur of the imaging optical system. Blurcorrection methods such as the maximum entropy method andRichardson-Lucy method have also been proposed in addition to thosefilters. They are equal to the Wiener filter in the sense thatprocessing is performed based on the OTF or PSF and the noise amount tocorrect a blur of a capturing image and obtain a corrected image,although a detailed description thereof will be omitted.

Generally, frequency components contained in a capturing image areenhanced in blur correction. Since a capturing image contains noise,blur correction enhances the frequency components of the noise as well.For this reason, if the degree of blur correction is strong, noise alsoincreases. To avoid the increase of noise, the degree of blur correctionneeds to be weakened. That is, the degree of blur correction and noiseincrease have a tradeoff relationship.

The Wiener filter is one of filters that implement optimum balancebetween the degree of blur correction and noise increase. Morespecifically, correction using a Wiener filter gives a corrected imageclosest in terms of square error to a capturing image containing neithernoise nor a blur by the imaging optical system. That is, the Wienerfilter performs correction in consideration of both noise increase andthe degree of blur correction and can therefore maintain appropriatebalance between noise increase and the degree of blur correction.

As described above, the Wiener filter can be calculated from the OTF andnoise amount. Conventionally, the Wiener filter is constructed based ona noise amount measured under a predetermined capturing condition or anempirically defined noise amount. However, the noise amount varies inaccordance with the capturing condition, and therefore, the noise amountmeasured under a predetermined capturing condition or the empiricallydefined noise amount is not accurate. As a result, the conventionalWiener filter does not necessarily have an optimum structure.

As an example of a noise amount variation according to the capturingcondition, the temperature of the image capturing device rises so as toincrease noise upon continuous shooting. A Wiener filter formed based ona noise amount smaller than the actual noise amount increases the noiseof a capturing image more than necessary. Conversely, a Wiener filterformed based on a noise amount larger than the actual noise amountweakens the degree of blur correction more than necessary. In otherwords, blur correction based on an incorrect noise amount leads toinsufficient correction or excessive noise.

In the above-described blur correction methods such as the maximumentropy method and Richardson-Lucy method as well, the degree of blurcorrection and noise increase have a tradeoff relationship. Hence, if noaccurate noise amount is obtained, optimum blur correction cannot bedone.

SUMMARY OF THE INVENTION

In one aspect, an image processing apparatus comprising: a firstacquisition section, configured to acquire a capturing parameter and acapturing image obtained by an image capturing device which uses thecapturing parameter; a second acquisition section, configured toacquire, corresponding to (a) an optical transfer function of the imagecapturing device derived from the capturing parameter, and (b) a noiseamount of the capturing image dependent on the capturing parameter,correction data to correct a blur of the capturing image, wherein afirst degree of correction by the correction data for a high noiseamount is less than a second degree of correction by the correction datafor a low noise amount.

According to the aspect, it is possible to acquire correction data forblur correction corresponding to the noise amount of a capturing image.It is also possible to perform blur correction while balancing noiseincrease and the degree of blur correction well.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the arrangement of an image capturingapparatus according to an embodiment.

FIG. 2 is a flowchart for explaining image processing of an imageprocessing unit.

FIG. 3 is a view for explaining an example of the characteristic of adark image.

FIG. 4 is a table showing the concept of a correction coefficientstorage unit.

FIGS. 5A to 5C are graphs schematically showing the relationship betweenthe noise amount and blur correction.

FIG. 6 is a flowchart for explaining processing of a blur correctionunit.

FIG. 7 is a block diagram showing the arrangement of an image capturingapparatus according to the second embodiment.

FIG. 8 is a flowchart for explaining processing of a blur correctionunit according to the second embodiment.

FIG. 9 is a view showing the concept of an OB portion.

FIG. 10 is a block diagram showing the arrangement of an image capturingapparatus according to the third embodiment.

FIG. 11 is a block diagram showing the arrangement of an image capturingapparatus according to the fourth embodiment.

FIG. 12 is a flowchart for explaining image processing of an imageprocessing unit according to the fourth embodiment.

FIG. 13 is a table showing the concept of a noise characteristic storageunit.

FIG. 14 is a block diagram showing the arrangement of an image capturingapparatus according to the fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

An image processing apparatus and image processing method according tothe embodiments of the present invention will now be described in detailwith reference to the accompanying drawings.

First Embodiment [Arrangement of Apparatus]

FIG. 1 is a block diagram showing the arrangement of an image capturingapparatus according to the embodiment.

Light from an object (not shown) passes through an imaging opticalsystem 101 and forms an image on the light-receiving surface of an imagecapturing device 102. The image capturing device 102 converts the lightthat forms an image on the light-receiving surface into an electricalsignal. An A/D converter (not shown) converts the electrical signal intoa digital signal, thereby obtaining a capturing (captured) image 104.

The image capturing device 102 is formed from a CCD or CMOS sensor whichconverts the optical signal of the image formed on the light-receivingsurface into an electrical signal for each of photoelectric conversionelements arranged on the light-receiving surface. A shade unit 103 has afunction of blocking off the light from the object not to make it reachthe light-receiving surface of the image capturing device 102. The shadeunit 103 may be, for example, a shutter or a stop if it can block offlight in the aperture state stopped down to its limit. Note that tocapture an image of an object, the shade unit 103 is kept open for apredetermined time. That is, light from the object is never entirelyblocked off during the capturing time.

A state detection unit 131 acquires capturing parameters, that is, aseries of parameters representing the capturing state of the imagecapturing apparatus. Examples of the capturing parameters are a lensidentifier, body identifier, stop value, object distance, zoom position,and ISO sensitivity. The capturing parameters specify the capturingcharacteristics (the characteristics of the imaging optical system 101and the characteristics of the image capturing device 102) in thecapturing state. That is, the capturing parameter varies in dependenceupon use of the imaging optical system. The capturing parametersrepresent real time (current) characteristics of the image capturingapparatus. Acquiring capturing parameters in this way facilitatesaccurate calculation of noise in a captured image and as a consequence,the captured image can be corrected in an accurate manner in order toremove image artifacts such as blur and noise. This is in contrast tothe conventionally known technique mentioned in the backgrounddiscussion, which uses predetermined capturing parameters to calculatenoise in a captured image. However, noise varies in dependence on thecapturing parameter and therefore noise calculated using a predeterminedcapturing parameter (or empirically defined noise) is not accurate.

In an image processing unit 121, a noise amount estimation unit 111 (tobe described later in detail) estimates a noise amount 112 using a darkimage 105. A preprocessing unit 106 (to be described later in detail)executes preprocessing of blur correction for the capturing image 104using the dark image 105. A blur correction unit 108 notifies acorrection coefficient interpolation unit 113 of a pixel position 114 ofa correction target, and corrects, using correction coefficients 115returned from the correction coefficient interpolation unit 113, theblur of the capturing image 104 input from the preprocessing unit 106.

Based on the noise amount 112 and a capturing parameter 107, thecorrection coefficient interpolation unit 113 acquires, from acorrection coefficient storage unit 116, the correction coefficients 115corresponding to the pixel position 114 sent from the blur correctionunit 108, and returns the correction coefficients 115 to the blurcorrection unit 108. A post-processing unit 119 performs post-processingsuch as color conversion and gain adjustment for the blur-correctedcapturing image input from the blur correction unit 108, and outputs acorrected image 110. An image recording unit 141 records the correctedimage 110 on a recording medium such as a memory card.

[Processing of Image Processing Unit]

FIG. 2 is a flowchart for explaining image processing of the imageprocessing unit 121.

The image processing unit 121 acquires the capturing image 104 (S201)and the dark image 105 (S202).

The dark image 105 is data obtained by causing the shade unit 103 toblock off light from the object and A/D-converting an electrical signaloutput from the image capturing device 102. The electrical signal outputfrom the image capturing device 102 normally contains noise. Even in theshaded state, the value of the electrical signal output from the imagecapturing device 102 is not zero. That is, not all the pixel values ofthe dark image 105 are zero, and the pixels have certain values. Inother words, noise defines the pixel values of the dark image 105. Thedark image 105 has information representing the noise characteristic ofthe image capturing device 102.

FIG. 3 is a view for explaining an example of the characteristic of adark image.

Reference numeral 306 denotes a light-receiving surface of the imagecapturing device 102, which corresponds to the dark image 105. Noise ofthe photoelectric conversion elements in a region 301 near a powersupply 304 tends to grow due to the heat of the power supply 304. Forthis reason, the pixel values in the region 301 of the dark image 105tend to be larger than the average pixel value of the overall dark image105.

When a flashlight 305 emits light, noise of the photoelectric conversionelements in a region 302 close to the flashlight 305 also grows due toheat. Hence, the pixel values in the region 302 of the dark image 105tend to be larger than the average pixel value of the overall dark image105.

A region 303 of the image capturing device 102 sometimes has a noisecharacteristic different from that in other regions because of pixeldefects or the like. That is, as a characteristic feature, the pixelvalues of the dark image 105 change between the regions.

The capturing image 104 also contains similar noise. However, it isdifficult to estimate the noise characteristic from the capturing image104 because the capturing image 104 contains both object information andnoise information. That is, acquiring the dark image 105 in addition tothe capturing image 104 of the object gives an estimate of the noiseamount in the capturing image 104.

Next, the image processing unit 121 acquires the capturing parameter 107from the state detection unit 131 (S203). When the capturing parameter107 is decided, the OTF of the imaging optical system 101 can beobtained by measurement or simulation. Since the OTF of the imagingoptical system 101 changes depending on the capturing parameter 107,acquisition of the capturing parameter 107 is indispensable to correctthe blur of the imaging optical system 101.

The image processing unit 121 then causes the noise amount estimationunit 111 to estimate the noise amount 112 from the dark image 105(S204), and causes the preprocessing unit 106 to execute preprocessing(S205) (to be described later in detail). The blur correction unit 108performs processing (blur correction) of reducing the blur of thecapturing image 104 after the preprocessing using the correctioncoefficients 115 acquired via the correction coefficient interpolationunit 113 (S206).

The image processing unit 121 causes the post-processing unit 119 toexecute post-processing for the blur-corrected capturing image (S207),and outputs the corrected image 110 after the post-processing to theimage recording unit 141 (S208) to record the corrected image 110 on arecording medium.

[Noise Amount Estimation Unit]

The noise amount estimation unit 111 divides the dark image 105 into aplurality of regions, and calculates the standard deviation of the pixelvalues in each region. In a region where the noise amount is large, thestandard deviation is large. In a region where the noise amount issmall, the standard deviation is small. That is, it is possible toaccurately grasp the noise amount that changes between the regions, asdescribed with reference to FIG. 3, by calculating the standarddeviation in each region.

When the ISO sensitivity is high, normally, processing of amplifying thepixel values of the capturing image 104 is performed. For this reason,the preprocessing unit 106 or post-processing unit 119 amplifies thepixel values of the capturing image 104 which has been obtained whilesetting the ISO sensitivity high. Noise contained in the capturing image104 is also amplified in accordance with the set ISO sensitivity. It istherefore impossible to simply use the standard deviation obtained fromthe dark image 105 intact as the noise amount of the capturing image104. Instead, the standard deviation obtained from the dark image 105 isamplified at a predetermined ratio corresponding to the set ISOsensitivity, and the amplification result is used as the actual noiseamount 112.

Note that the standard deviation is used as the noise amount 112 in theabove-described example. A pixel value variance or the differencebetween the maximum value and the minimum value can also serve as anindicator representing the magnitude of noise because they become largeras the noise increases and smaller as the noise decreases. Hence, notthe standard deviation but the variance or the difference between themaximum value and the minimum value may be used as the noise amount 112.

[Correction Coefficient Interpolation Unit]

The correction coefficient interpolation unit 113 acquires thecorrection coefficients 115 for blur correction corresponding to thenoise amount 112, capturing parameter 107, and pixel position 114. Thenoise amount 112, capturing parameter 107, and pixel position 114 areinformation necessary for acquiring the correction coefficients 115. Anexample will be explained below in which blur correction is performedusing a Wiener filter. Note that in blur correction using a Wienerfilter, the correction coefficients 115 are the coefficients of theWiener filter.

FIG. 4 is a table showing the concept of the correction coefficientstorage unit 116.

The correction coefficient storage unit 116 stores, in advance, thecorrection coefficients 115 (Wiener filter) corresponding to eachcombination of the capturing parameter 107, pixel position 114, andnoise amount 112. FIG. 4 shows a stop value and a zoom position asexamples of the capturing parameter 107. The OTF changes depending onthe pixel position 114. When the OTF changes, the correctioncoefficients 115 also change. The correction coefficients 115 alsodepend on the noise amount 112.

The Wiener filter is a two-dimensional 3×3 digital filter, as indicatedby the correction coefficients in FIG. 4. The Wiener filter formingmethod will be described later in detail.

There are an enormous number of combinations of the capturing parameter107, pixel position 114, and noise amount 112. If the correctioncoefficients 115 corresponding to all combinations are stored, the dataamount stored in the correction coefficient storage unit 116 becomesenormous. To prevent this, the correction coefficient storage unit 116stores correction coefficients corresponding to the representativevalues of the capturing parameter 107, pixel position 114, and noiseamount 112. Hence, the correction coefficient interpolation unit 113acquires, from the correction coefficient storage unit 116, correctioncoefficients corresponding to representative values adjacent to thecombination of the input capturing parameter 107, pixel position 114,and noise amount 112. Based on the correction coefficients, thecorrection coefficient interpolation unit 113 interpolates thecorrection coefficients 115 corresponding to the combination of thecapturing parameter 107, pixel position 114, and noise amount 112.

For example, the representative values of the pixel position 114 are setat two points, i.e., an edge of the image and the center of the image.The correction coefficients 115 corresponding to arbitrary pixelpositions 114 are approximately obtained by calculating the weightedaverage of the correction coefficients at the edge of the image and thatat the center of the image.

The Wiener filter has been exemplified above. If blur correction such asthe maximum entropy method or Richardson-Lucy method is used, thecorrection coefficients 115 correspond to the OTF and noise amount.

[Wiener Filter Forming Method]

The Wiener filter can be obtained by calculating the inverse Fouriertransform of W given by

W(u,v)=H*(u,v)/{|H(u,v)|² +Sn(u,v)/Sf(u,v)}  (5)

where

H*(u,v) is the complex conjugate of H(u,v) that is the OTF,

Sn(u,v) is the power spectrum of noise, and

Sf(u,v) is the power spectrum of the capturing image.

The Wiener filter is also given, by rewriting Sn(u,v) and Sf(u,v) ofequation (5) into a simplified form free from dependence on the spatialfrequency, by

W(u,v)=H*(u,v)/{|H(u,v)|² +SNR ²}  (6)

where SNR is the SN ratio obtained by dividing the standard deviation ofnoise by the pixel value of the capturing image.

When calculating the standard deviation of noise in each region of thedark image 105, the pixel value to be used for division is the averagepixel value in the corresponding region of the capturing image 104.Strictly speaking, the SNR is a quantity which depends on the pixelvalues of the correction target pixels. However, the average pixel valueof the capturing image 104 may be used as a matter of expedience.

[Relationship between Noise Amount and Blur Correction]

FIGS. 5A to 5C are graphs schematically showing the relationship betweenthe noise amount and blur correction. The ordinate of each graphrepresents an MTF (Modulation Transfer Function) that is the absolutevalue of the OTF, and the abscissa represents the spatial frequency.Note that in FIGS. 5A to 5C, a point having a significant value otherthan 0 is regarded as an object (point source) for the descriptiveconvenience.

FIG. 5A shows graphs for explaining the MTF when no noise is present.The left graph of FIG. 5A shows the MTF of an object. If the imagingoptical system 101 has no blur, the capturing image of the objectexhibits the same MTF. The middle graph of FIG. 5A shows the MTF of acapturing image including the blur of the imaging optical system 101.The higher the frequency is, the smaller the value of the MTF is. Notethat as the object is assumed to be a point source, the middle graph ofFIG. 5A represents the MTF of the imaging optical system 101, too.

The right graph of FIG. 5A shows the MTF of a capturing image obtainedby performing blur correction using an inverse filter for a capturingimage corresponding to the middle graph of FIG. 5A. Within the range ofMTF>0 in the middle graph of FIG. 5A, blur correction using the inversefilter can recover the capturing image to the same state as in the rightgraph of FIG. 5A. In fact, however, noise is present, and the frequencycharacteristic of noise is added to the MTF of the capturing image.

FIG. 5B shows graphs for explaining the MTF when noise is present, andaccurate noise amount estimation is possible. A region 51 a in the leftgraph of FIG. 5B represents the noise component added to the MTF of thecapturing image. The middle graph of FIG. 5B shows the MTF of acapturing image obtained by performing blur correction using an inversefilter for a capturing image corresponding to the left graph of FIG. 5B.As indicated by a region 51 b, the noise extremely increases in thehigh-frequency region.

The right graph of FIG. 5B shows the MTF of a capturing image obtainedby performing blur correction using a Wiener filter for a capturingimage corresponding to the left graph of FIG. 5B. The Wiener filterreduces the degree of correction in a frequency region where the ratioof the noise component to the MTF of the imaging optical system 101rises. As a result, the degree of correction lowers in thehigh-frequency region where the ratio of the noise component is high. Asindicated by a region 51 c in the right graph of FIG. 5B, the increaseof noise is suppressed.

In the low- and intermediate-frequency regions, the MTF shown in theright graph of FIG. 5B is nearly 1, and the blur is corrected in thesefrequency regions. That is, blur correction that obtains a capturingimage corresponding to the MTF shown in the right graph of FIG. 5B ispreferable because it moderately balances the effect of blur correctionand the adverse effect of noise increase. To obtain such balance, it isnecessary to accurately estimate the noise amount.

FIG. 5C shows graphs for explaining the MTF when noise is present, andaccurate noise amount estimation is impossible. The left graph of FIG.5C shows the MTF of a capturing image obtained by performing blurcorrection using a Wiener filter for a capturing image corresponding tothe left graph of FIG. 5B. In this graph, the noise amount is estimatedto be smaller than the actual amount. Since the noise amount isestimated to be smaller than the actual amount, the noise increases inthe high-frequency region, as indicated by a region 52 a.

The right graph of FIG. 5C shows the MTF of a capturing image obtainedby performing blur correction using a Wiener filter for a capturingimage corresponding to the left graph of FIG. 5B. In this graph, thenoise amount is estimated to be larger than the actual amount. Since thenoise amount is estimated to be larger than the actual amount, the MTFin the intermediate-frequency region lowers, and blur correction isinsufficient, although the increase of noise is suppressed in thehigh-frequency region, as indicated by a region 52 b.

[Preprocessing Unit]

The preprocessing unit 106 performs, for the capturing image 104, forexample, gamma correction, demosicing (developing process), andprocessing of compensating for defects (pixel omission) in the imagecapturing device 102 as needed. The preprocessing unit 106 also executespreprocessing based on the dark image 105 for the capturing image 104.Although preprocessing based on the dark image 105 is not directlyrelevant to blur correction, it will be described as an example of usingthe dark image 105 for processing other than estimation of the noiseamount 112.

When noise exists, a region of the capturing image 104 corresponding tothe darkest region of the object also has a significant value other than0. Consequently, the contrast lowers in the capturing image 104. Thecontrast decrease is reduced by subtracting the average pixel value ofthe dark image 105 from each pixel value of the capturing image 104. Ifthe noise characteristic largely changes between regions, as shown inFIG. 3, the average pixel value is calculated in each region of the darkimage 105 and subtracted from the pixel values in a corresponding regionof the capturing image 104. If the noise characteristic changesdepending on the pixel level, the dark image 105 itself is subtractedfrom the capturing image 104. Especially when a defective pixel thatnever takes the value 0 exists, it is preferable to subtract the darkimage 105 from the capturing image 104.

As another preprocessing, noise reduction may be done based on the darkimage 105. For example, a region with large noise is determined in thedark image 105, and stronger noise reduction is applied to acorresponding region of the capturing image 104. This enables noisereduction adaptive to the actual noise characteristic and enhances theeffect of noise reduction of the capturing image 104. Since thecapturing image 104 after noise reduction contains less noise, thepreprocessing unit 106 needs to notify the noise amount estimation unit111 of information about the reduced noise amount. The noise amountestimation unit 111 needs to correct the noise amount 112 based on thenotification.

[Blur Correction Unit]

When executing blur correction using a Wiener filter, the blurcorrection unit 108 performs a convolution operation of the Wienerfilter (correction coefficients 115) and the capturing image 104. Whenexecuting blur correction using a blur correction method such as themaximum entropy method or Richardson-Lucy method, a repetitive operationis performed using an OTF and noise amount corresponding to thecorrection coefficients 115, thereby correcting the blur of thecapturing image 104.

FIG. 6 is a flowchart for explaining processing of the blur correctionunit 108.

The blur correction unit 108 outputs the position of a pixel of interestof the capturing image 104 as the pixel position 114 (S101), andacquires the Wiener filter (correction coefficients 115) from thecorrection coefficient interpolation unit 113 (S102). The blurcorrection unit 108 then performs the convolution operation of theWiener filter and 3×3 pixels centering around the pixel of interest(S103), and outputs a corrected pixel corresponding to the pixel ofinterest (S104).

Next, the blur correction unit 108 determines whether all pixels of thecapturing image 104 have been processed (S105). If the processing hasnot ended yet, the pixel of interest is moved (S106), and the processreturns to step S101. If all pixels have been processed, the blurcorrection ends.

In the above-described way, using the dark image 105 gives an accurateestimate of the noise amount 112 that varies depending on the capturingcondition. When blur correction is performed using the accuratelyestimated noise amount 112 and the correction coefficients 115associated with the blur characteristic of the imaging optical system101, noise increase and the degree of blur correction can balance well.As a result, the blur caused by the imaging optical system 101decreases, and the corrected image 110 in which noise increase ismoderately suppressed can be obtained.

The degree of correction of each frequency component included in thecapturing image 104 changes in accordance with the pixel position andthe capturing condition. For example, at a pixel position where thenoise amount is large or under a capturing condition with a high ISOsensitivity, the noise amount is large. As a result, the degree ofcorrection of each frequency component of the capturing image 104lowers. In this embodiment, the degree of correction of each frequencycomponent is adaptively changed for a different noise characteristic inaccordance with the pixel position and the capturing condition, therebymaintaining the balance between noise and the degree of blur correction.

Second Embodiment

Image processing according to the second embodiment of the presentinvention will be described below. Note that the same reference numeralsas in the first embodiment denote the same parts in the secondembodiment, and a detailed description thereof will not be repeated.

Shot noise is known as a noise generated when light strikes thelight-receiving surface of an image capturing device 102. The shot noiseis known to be proportional to the square root of light energy thatenters photoelectric conversion elements.

The noise amount estimation unit 111 of the first embodiment calculatesthe noise amount 112 based on the dark image 105 and the capturingparameter 107. That is, the noise amount 112 of the first embodimentrepresents a noise amount when no light strikes the light-receivingsurface of the image capturing device 102. In other words, the darkimage 105 captured in a light block off state contains no informationabout shot noise.

In the second embodiment, a high noise estimation accuracy is attainedin consideration of shot noise.

FIG. 7 is a block diagram showing the arrangement of an image capturingapparatus according to the second embodiment. Unlike the arrangement ofthe first embodiment shown in FIG. 1, a noise amount estimation unit 111calculates a noise amount 112 based on a pixel value 118 of a capturingimage 104 and information from a shot noise characteristic storage unit117 in addition to a dark image 105 and a capturing parameter 107. Thenoise amount estimation unit 111 of the second embodiment estimates thenoise amount 112 by

STDt=√/(STDd ² +STDs ²)  (7)

where

STDt is the estimated noise amount,

STDd is the noise amount calculated from the dark image 105, and

STDs is the shot noise amount.

STDd is the standard deviation of the dark image 105, as in the firstembodiment. STDs is determined by light energy. STDs can be obtained bycalculating pixel values because the pixel values are determined bylight energy. More specifically, the shot noise characteristic storageunit 117 stores a lookup table (LUT) representing the relationshipbetween a pixel value and STDs, and STDs is obtained based on the pixelvalue 118 of a pixel of interest obtained from a blur correction unit108. Note that STDt (noise amount 112) needs to be adjusted inaccordance with the ISO sensitivity setting, as in the first embodiment.

The LUT of the shot noise characteristic storage unit 117 is created inthe following way. An object having uniform brightness is captured. Thecorrespondence between the shot noise amount and the pixel value iscalculated based on the pixel values of the capturing image and thestandard deviation of the pixel values, thereby generating a LUT.

FIG. 8 is a flowchart for explaining processing of the blur correctionunit 108 according to the second embodiment. The blur correction unit108 outputs the value of a pixel of interest of the capturing image 104as the pixel value 118 (S100), and from then on, executes the sameprocesses (S101 to S106) as in the first embodiment.

For example, the pixel value of a bright pixel is larger than noise, andthe SNR is high. When the SNR is high, the degree of blur correction bya Wiener filter strengthens. Hence, a preferable result is obtained suchthat stronger blur correction is performed for a bright pixel. A pixelvalue in a dark portion of an image where noise is noticeable is notmuch larger than noise and has a low SNR. When the SNR is low, thedegree of blur correction by a Wiener filter weakens. Hence, apreferable result of suppressing noise increase in the dark portion ofthe image is obtained.

According to the second embodiment, since the noise amount 112 changesin accordance with the pixel value 118, the Wiener filter may frequentlychange. If changing the Wiener filter is cumbersome, for example, athreshold may be set. When the pixel value exceeds this threshold, thepredetermined pixel value 118 is output, thereby suppressing thefrequent change of the Wiener filter in a bright portion of an imagewith a high SNR.

In the second embodiment, considering the shot noise characteristicstorage unit 117, a memory of a larger storage capacity is necessary, ascompared to the first embodiment. For this reason, the arrangement ofthe second embodiment is not suitable for an image capturing apparatusin which the shot noise amount is smaller than the noise amount obtainedfrom the dark image 105. On the other hand, the arrangement is suitablefor an image capturing apparatus in which the shot noise amount islarger than the noise amount obtained from the dark image 105.

Third Embodiment

Image processing according to the third embodiment of the presentinvention will be described below. Note that the same reference numeralsas in the first and second embodiments denote the same parts in thethird embodiment, and a detailed description thereof will not berepeated.

There exist commercially available image capturing apparatuses such asan electronic camera which records and plays back a still image or amoving image captured by a solid-state image capturing device such as aCCD or CMOS sensor while using, as a recording medium, a memory cardhaving a solid-state memory device. Many of the image capturing devicesof these electronic cameras include a plurality of pixels shaded by analuminum film or the like and called OB (Optical Black) pixels. Imagedata output from the OB pixel range (to be referred to as an OB portionhereinafter) is called OB data.

FIG. 9 is a view showing the concept of an OB portion. An OB portion 801is arranged adjacent to a capturing region 802 and used to define blackof a capturing image. OB data has a significant value exceeding 0 due tonoise. The average value of the OB data also exceeds 0. Considering thatthe same noise as the OB data is mixed in the capturing image as well,the pixel having the minimum value in the capturing image has not thevalue 0 but a value equal to or larger than the average value of OBdata. Processing of subtracting the average value of OB data from acapturing image to correct the value of each pixel unexposed to light toalmost 0 is known. This processing will be referred to as a blacksubtraction process hereinafter. Light to the OB portion 801 is alreadyblocked off. For this reason, the use of the OB portion 801 makes theabove-described shade unit 103 not always necessary.

FIG. 10 is a block diagram showing the arrangement of an image capturingapparatus according to the third embodiment. Unlike the first embodimentshown in FIG. 1, the apparatus includes no shade unit 103, apreprocessing unit 106 performs the black subtraction process ofsubtracting the average value of OB data 120 from a capturing image 104as one of preprocesses, and a noise amount estimation unit 111 estimatesa noise amount 112 based on the OB data 120.

That is, an image processing unit 121 of the third embodiment uses theOB data 120 not only for the black subtraction process but also forestimation of the noise amount 112. More specifically, the noise amountestimation unit 111 uses the standard deviation of the OB data 120 asthe noise amount 112.

Needless to say, the noise amount estimation unit 111 may estimate thenoise amount in consideration of shot noise, as in the secondembodiment. In this case, not all pixels of the OB portion 801 need beused. A noise amount STDd may be calculated from the OB data 120 of somepixels.

In the first and second embodiments, to estimate the noise amount 112,it is necessary to perform capturing for obtaining the capturing image104 and capturing for obtaining the dark image 105 in a light block offstate. In the third embodiment, the OB data 120 can be acquired from theOB portion 801 aside from capturing of the capturing image 104. Thissimplifies the processing procedure. However, since the OB portion 801is located in a region different from the capturing region 802, it isimpossible to accurately estimate the noise amount that changes betweenregions of the capturing image 104. In other words, the arrangement ofthe third embodiment is suitable for an image capturing apparatus inwhich the degree of noise amount change between regions of the capturingimage 104 is low. However, the third and fourth embodiments can becombined.

Fourth Embodiment

Image processing according to the fourth embodiment of the presentinvention will be described below. Note that the same reference numeralsas in the first to third embodiments denote the same parts in the fourthembodiment, and a detailed description thereof will not be repeated.

FIG. 11 is a block diagram showing the arrangement of an image capturingapparatus according to the fourth embodiment. Unlike the firstembodiment shown in FIG. 1, the apparatus includes no shade unit 103, apreprocessing unit 106 performs a black subtraction process of acapturing image 104 based on a noise amount 112 as one of preprocesses,and the apparatus includes a noise characteristic storage unit 122 forestimation of the noise amount 112. The noise characteristic storageunit 122 stores the correspondence between a capturing parameter 107 andthe noise amount 112. Hence, a noise amount estimation unit 111 acquiresthe noise amount 112 corresponding to the capturing parameter 107 fromthe noise characteristic storage unit 122.

Examples of the capturing parameter 107 associated with the noise amount112 are the temperature, exposure time, and ISO sensitivity setting ofan image capturing device 102. When estimating the noise amount 112 inconsideration of shot noise, as in the second embodiment, the capturingparameter 107 needs to include the exposure amount or pixel value.

FIG. 12 is a flowchart for explaining image processing of an imageprocessing unit 121 according to the fourth embodiment. The processingdoes not include step S202 of acquiring a dark image 105, unlike thefirst embodiment shown in FIG. 2. FIG. 13 is a table showing the conceptof the noise characteristic storage unit 122.

The noise characteristic storage unit 122 stores, as a LUT in advance,noise amounts corresponding to combinations of the capturing parameters107, for example, combinations of the temperature, exposure amount,exposure time, and ISO sensitivity set value of the image capturingdevice 102. To create the LUT to be stored in the noise characteristicstorage unit 122, the noise amount is measured in darkness for eachcombination of the capturing parameters 107.

When measuring the noise amount, if the value of each item of thecapturing parameters 107 is set in small steps, the number of times ofmeasurement is enormous, and the capacity necessary for storing the LUTalso increases enormously. To prevent this, the step of the value ofeach item is preferably adjusted in accordance with the storage capacityof the memory mountable in the image capturing apparatus. For example,only the ISO sensitivity setting may be changed, and the remaining itemsmay be fixed to predetermined values, or the noise amount measuringpoints may be limited to two. If the LUT of the noise characteristicstorage unit 122 records no noise amount corresponding to thecombination of the capturing parameters 107, the noise amount estimationunit 111 acquires a noise amount corresponding to a combination adjacentto the combination of the capturing parameters 107. The noise amount 112corresponding to the combination of the capturing parameters 107 isinterpolated from the noise amounts.

According to the fourth embodiment, a storage capacity for the noisecharacteristic value is necessary. However, since the noise amount 112need not be calculated based on the dark image 105 or OB data 120, thecalculation cost can be reduced.

Fifth Embodiment

Image processing according to the fifth embodiment of the presentinvention will be described below. Note that the same reference numeralsas in the first to fourth embodiments denote the same parts in the fifthembodiment, and a detailed description thereof will not be repeated.

FIG. 14 is a block diagram showing the arrangement of an image capturingapparatus according to the fifth embodiment. Unlike the fourthembodiment shown in FIG. 12, the apparatus includes no noisecharacteristic storage unit 122. A noise amount estimation unit 111 ofthe fifth embodiment holds a formula for calculating a noise amount 112from capturing parameters 107.

The formula held by the noise amount estimation unit 111 is obtained bymodeling the relationship between the noise amount and a combination ofthe capturing parameters 107 shown in FIG. 13 using regression analysisor the like. Note that the most simple formula generating method is amethod using repression analysis. However, for example, a physical modelformula of noise may be used.

According to the fifth embodiment, the storage capacity for the noisecharacteristic value can be decreased as compared to the fourthembodiment. However, since the noise characteristic is approximated bythe formula, the accuracy of estimating the noise amount 112 may lowerdue to an approximation error, as compared to the fourth embodiment.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2009-104548, filed Apr. 22, 2009, which is hereby incorporated byreference herein in its entirety.

1.-11. (canceled)
 12. An image processing apparatus comprising: a firstacquisition section, configured to acquire a capturing parameter and acapturing image obtained by an image capturing device which uses thecapturing parameter; and a second acquisition section, configured toacquire, corresponding to (a) an optical transfer function of the imagecapturing device derived from the capturing parameter, and (b) an ISOsensitivity included in the capturing parameter, correction data tocorrect a blur of the capturing image.
 13. The apparatus according toclaim 12, wherein a first degree of correction by the correction datafor a high ISO sensitivity is less than a second degree of correction bythe correction data for a low ISO sensitivity.
 14. The apparatusaccording to claim 12, further comprising a corrector configured toperform, for the capturing image, blur correction based on thecorrection data.
 15. The apparatus according to claim 12, wherein thesecond acquisition section acquires, corresponding to (a) the opticaltransfer function of the image capturing device derived from thecapturing parameter, and (b) the ISO sensitivity included in thecapturing parameter, the correction data for each pixel of the capturingimage to correct the blur of the capturing image.
 16. The apparatusaccording to claim 12, wherein the second acquisition section acquires,corresponding to (a) the optical transfer function of the imagecapturing device derived from the capturing parameter, and (b) the ISOsensitivity included in the capturing parameter, the correction data tocorrect the blur of the capturing image using a calculator.
 17. Theapparatus according to claim 16, further comprising an input sectionconfigured to input dark image data output from the image capturingdevice in a state in which no light is input to the image capturingdevice, wherein the second acquisition section calculates a noise amountcontained in the capturing image by correcting, based on the ISOsensitivity, a noise amount acquired from the dark image data, andwherein a first degree of correction by the correction data for a highnoise amount is less than a second degree of correction by thecorrection data for a low noise amount.
 18. The apparatus according toclaim 16, further comprising: an input section, configured to input darkimage data output from the image capturing device in a state in which nolight is input to the image capturing device; and a memory which storesa noise amount generated when light strikes a light-receiving surface ofthe image capturing device, wherein the second acquisition sectioncalculates a noise amount contained in the capturing image from a noiseamount acquired from the dark image data, the ISO sensitivity, and anoise amount acquired from the memory based on a pixel value of a pixelof interest, and wherein a first degree of correction by the correctiondata for a high noise amount is less than a second degree of correctionby the correction data for a low noise amount.
 19. The apparatusaccording to claim 16, further comprising an input section configured toinput image data output from a shaded pixel arranged in the imagecapturing device, wherein the second acquisition section calculates anoise amount contained in the capturing image by correcting, based onthe ISO sensitivity, a noise amount acquired from the image data, andwherein a first degree of correction by the correction data for a highnoise amount is less than a second degree of correction by thecorrection data for a low noise amount.
 20. The apparatus according toclaim 12, further comprising a memory which stores correspondencebetween the capturing parameter and a noise amount, wherein the secondacquisition section acquires a noise amount contained in the capturingimage from the memory based on the ISO sensitivity, and acquires thecorrection data for each pixel of the capturing image, and wherein afirst degree of correction by the correction data for a high noiseamount is less than a second degree of correction by the correction datafor a low noise amount.
 21. The apparatus according to claim 12, whereinthe second acquisition section acquires a noise amount contained in eachof pixels of interest of the capturing image by calculation based on theISO sensitivity, and acquires the correction data for each pixels of thecapturing image, wherein a first degree of correction by the correctiondata for a high noise amount is less than a second degree of correctionby the correction data for a low noise amount.
 22. An image processingmethod performed by a processor, the method comprising: using theprocessor to perform the steps of: acquiring a capturing parameter and acapturing image obtained by an image capturing device which uses thecapturing parameter; and acquiring correction data, corresponding to (a)an optical transfer function of the image capturing device derived fromthe capturing parameter, and (b) an ISO sensitivity included in thecapturing parameter, to correct a blur of the capturing image.
 23. Anon-transitory computer-readable medium storing a computer-executableprogram for causing a computer to perform an image processing method,the method comprising the steps of: acquiring a capturing parameter anda capturing image obtained by an image capturing device which uses thecapturing parameter; and acquiring correction data, corresponding to (a)an optical transfer function of the image capturing device derived fromthe capturing parameter, and (b) an ISO sensitivity included in thecapturing parameter, to correct a blur of the capturing image.